In many newscasts the studio set is not a real set, but is a virtual set rendered in a computer. The newscaster or other foreground subject is inserted by an image compositing process commonly referred to as chroma key or matting. This process requires a colored backing, typically blue (or green), which is replaced in the compositing process by the virtual set. The actual colored backing is of limited size, and on wide-angle shots, or when the camera is panned or tilted too far, the camera field is likely to extend beyond the backing revealing the studio walls, ceiling, lights, etc. It is therefore necessary to extend the backing electronically be generating what are known as windows or garbage mattes, which cover or replace the exposed stage walls. Such generated electronic windows must not show a visible line where they join the actual colored backing.
The backing typically includes a vertical wall and a horizontal floor. The shadow cast by a subject on the backing floor darkens the floor in the area of the shadow. A control signal proportional to the luminance of the colored backing is used to enable the background scene i.e., the virtual set. The reduced luminance in the shadow area reduces the level of the control signal thus retaining the shadow in the composite.
The linearity of the control signal which permits the retention of shadows also causes visibility of the backing-to-window join line when the backing luminance is not perfectly uniform. Backings that are front illuminated with multiple overlapping light sources cannot be illuminated with perfect uniformity. Because the background scene is enabled in its respective areas by a non uniform physical backing, and a uniform electronic window, the residual luminance nonuniformity of the backing causes a visible discontinuity in the background scene where the electronic window joins the physical backing. Current methods for eliminating the visibility of the join line in image compositing systems usually result in a visible loss of edge detail and the loss of shadows.
A method for backing luminance and color non uniformity compensation in image compositing systems (commonly referred to as `screen correction`), which eliminates the window join line and does not degrade the image, is described in U.S. Pat. No. 5,424,781. A complete description of this method is provided in the referenced patent. In summary, the RGB levels of every pixel on the colored backing is compared to a reference point on the backing and their RGB difference is added to the backing RGB levels to make it perfectly uniform in color and luminance. An inhibit signal is generated to prevent the corrections from being applied to the subject. A product utilizing this screen correction method is the "ultimatte-8" available from Ultimatte Corporation.
A clear frame is required by the above referenced patent for the generation of the inhibit signal. A clear frame is a frame of the colored backing in which all subjects have been cleared from the set. In a non live broadcast environment, screen correction `clear frames` are generated by repeating the camera moves with a motion control system, with the talent (i.e., foreground subject) removed, and recording the cleared set. In a live broadcast this is not possible.
In order to use the screen correction process in the above referenced patent, it is necessary to generate, in real time (30 or 25 frames/sec.) a synthetic clear frame for each individual image frame. These clear frames, in conjunction with screen correction as described in the above referenced patent, make window edges invisible.
U.S. Pat. No. 5,515,109 describes a method of generating synthetic clear frames by manual identification of subjects and their shadows. This method is not practical for live broadcasts because manual identification of shadow areas cannot occur in real time.
The essence of this invention is to produce a series of synthetic clear frames, in real time, to permit the concurrent use of the screen correction technique of U.S. Pat. No. 5,424,781. The invention described here generates a synthetic clear frame for each image frame and thus permits the referenced screen correction method to function in live broadcasts.
The generation of a clear frame requires identifying those pixels that represent the subject as opposed to those pixels that represent the colored backing so that the subject may be removed. This is done by generating a control signal Ec, and processed video signals (PRV) of the foreground scene which permits identification of these areas.
The "Comprehensive Electronic Compositing System" of U.S. Pat. No. 4,344,085, for example, develops a control signal "Ec", and "Processed Video" (PRV) signal. Control signal Ec is proportional to the luminance and visibility of the backing, and controls the level of the background scene. This Ec matte equation has the form of EQU Ec=B-max (G,R)
for a blue backing
where B is the blue signal level in an RGB signal and max (G,R) is the higher of the green and red signal levels of the RGB signal.
As shown in FIG. 1, the Ec matte is displayed as black silhouette 2 of the opaque subject against a white field 3. Subsequent patents assigned to Ultimatte Corporation describe expanded forms of the matte equation, and any one of these equations could be substituted for the basic equation above. Any control signal equation that permits identification of the unobstructed backing would function in generating a clear frame by the method described here.
The control signal, derived from the luminance of the backing, is subtracted from the RGB components in the colored backing in amounts sufficient to reduce each component just to zero. This "Processed Video" (PRV) signal displays the subject in full color against a black field. These two signals, Ec and PRV, are used to define and identify the subject area, the colored backing area, and the subject-to-backing transition area which is semitransparent. These areas are identified when the following relationships exist:
in the unobscured backing area, Ec&gt;0, and PRV=0 PA1 in a semitransparent area, Ec&gt;0, and PRV&gt;0 PA1 in the opaque subject area, Ec=0, and PRV.gtoreq.
These definitions are more fully explained in co-pending application Ser. No. 08/240,881, filed May 9, 1994, now U.S. Pat. No. 5,557,339.